Challenges of COVID-19 Case Forecasting in the US, 2020-2021 (preprint)

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naive baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making. Author SummaryAs SARS-CoV-2 began to spread throughout the world in early 2020, modelers played a critical role in predicting how the epidemic could take shape. Short-term forecasts of epidemic outcomes (for example, infections, cases, hospitalizations, or deaths) provided useful information to support pandemic planning, resource allocation, and intervention. Yet, infectious disease forecasting is still a nascent science, and the reliability of different types of forecasts is unclear. We retrospectively evaluated COVID-19 case forecasts, which were often unreliable. For example, forecasts did not anticipate the speed of increase in cases in early winter 2020. This analysis provides insights on specific problems that could be addressed in future research to improve forecasts and their use. Identifying the strengths and weaknesses of forecasts is critical to improving forecasting for current and future public health responses.

A Deep Learning Approach to Forecast Short-Term COVID-19 Cases and Deaths in the US (preprint)

Since the US reported its first COVID-19 case on January 21, 2020, the science community has been applying various techniques to forecast incident cases and deaths. To date, providing an accurate and robust forecast at a high spatial resolution has proved challenging, even in the short term. Here we present a novel multi-stage deep learning model to forecast the number of COVID-19 cases and deaths for each US state at a weekly level for a forecast horizon of 1 to 4 weeks. The model is heavily data driven, and relies on epidemiological, mobility, survey, climate, and demographic. We further present results from a case study that incorporates SARS-CoV-2 genomic data (i.e. variant cases) to demonstrate the value of incorporating variant cases data into model forecast tools. We implement a rigorous and robust evaluation of our model – specifically we report on weekly performance over a one-year period based on multiple error metrics, and explicitly assess how our model performance varies over space, chronological time, and different outbreak phases. The proposed model is shown to consistently outperform the CDC ensemble model for all evaluation metrics in multiple spatiotemporal settings, especially for the longer-term (3 and 4 weeks ahead) forecast horizon. Our case study also highlights the potential value of virus genomic data for use in short-term forecasting to identify forthcoming surges driven by new variants. Based on our findings, the proposed forecasting framework improves upon the available forecasting tools currently used to support public health decision making with respect to COVID-19 risk. Research in context Evidence before this study A systematic review of the COVID-19 forecasting and the EPIFORGE 2020 guidelines reveal the lack of consistency, reproducibility, comparability, and quality in the current COVID-19 forecasting literature. To provide an updated survey of the literature, we carried out our literature search on Google Scholar, PubMed, and medRxi , using the terms “Covid-19,” “SARS-CoV-2,” “coronavirus,” “short-term,” “forecasting,” and “genomic surveillance.” Although the literature includes a significant number of papers, it remains lacking with respect to rigorous model evaluation, interpretability and translation. Furthermore, while SARS-CoV-2 genomic surveillance is emerging as a vital necessity to fight COVID-19 (i.e. wastewater sampling and airport screening), to our knowledge, no published forecasting model has illustrated the value of virus genomic data for informing future outbreaks. Added value of this study We propose a multi-stage deep learning model to forecast COVID-19 cases and deaths with a horizon window of four weeks. The data driven model relies on a comprehensive set of input features, including epidemiological, mobility, behavioral survey, climate, and demographic. We present a robust evaluation framework to systematically assess the model performance over a one-year time span, and using multiple error metrics. This rigorous evaluation framework reveals how the predictive accuracy varies over chronological time, space, and outbreak phase. Further, a comparative analysis against the CDC ensemble, the best performing model in the COVID-19 ForecastHub, shows the model to consistently outperform the CDC ensemble for all evaluation metrics in multiple spatiotemporal settings, especially for the longer forecasting windows. We also conduct a feature analysis, and show that the role of explanatory features changes over time. Specifically, we note a changing role of climate variables on model performance in the latter half of the study period. Lastly, we present a case study that reveals how incorporating SARS-CoV-2 genomic surveillance data may improve forecasting accuracy compared to a model without variant cases data. Implications of all the available evidence Results: from the robust evaluation analysis highlight extreme model performance variability over time and space, and suggest that forecasting models should be accompanied with specifications on the conditions under which they perform best (and worst), in order to maximize their value and utility in aiding public health decision making. The feature analysis reveals the complex and changing role of factors contributing to COVID-19 transmission over time, and suggests a possible seasonality effect of climate on COVID-19 spread, but only after August 2021. Finally, the case study highlights the added value of using genomic surveillance data in short-term epidemiological forecasting models, especially during the early stage of new variant introductions.

An Evaluation of Prospective COVID-19 Modeling: From Data to Science Translation (preprint)

Background Infectious disease modeling can serve as a powerful tool for science-based management of outbreaks, providing situational awareness and decision support for policy makers. Predictive modeling of an emerging disease is challenging due to limited knowledge on its epidemiological characteristics. For COVID-19, the prediction difficulty was further compounded by continuously changing policies, varying behavioral responses, poor availability and quality of crucial datasets, and the variable influence of different factors as the pandemic progresses. Due to these challenges, predictive modeling for COVID-19 has earned a mixed track record. Methods We provide a systematic review of prospective, data-driven modeling studies on population-level dynamics of COVID-19 in the US and conduct a quantitative assessment on crucial elements of modeling, with a focus on the aspects of modeling that are critical to make them useful for decision-makers. For each study, we documented the forecasting window, methodology, prediction target, datasets used, geographic resolution, whether they expressed quantitative uncertainty, the type of performance evaluation, and stated limitations. We present statistics for each category and discuss their distribution across the set of studies considered. We also address differences in these model features based on fields of study. Findings Our initial search yielded 2,420 papers, of which 119 published papers and 17 preprints were included after screening. The most common datasets relied upon for COVID-19 modeling were counts of cases (93%) and deaths (62%), followed by mobility (26%), demographics (25%), hospitalizations (12%), and policy (12%). Our set of papers contained a roughly equal number of short-term (46%) and long-term (60%) predictions (defined as a prediction horizon longer than 4 weeks) and statistical (43%) versus compartmental (47%) methodologies. The target variables used were predominantly cases (89%), deaths (52%), hospitalizations (10%), and R_t (9%). We found that half of the papers in our analysis did not express quantitative uncertainty (50%). Among short-term prediction models, which can be fairly evaluated against truth data, 25% did not conduct any performance evaluation, and most papers were not evaluated over a timespan that includes varying epidemiological dynamics. The main categories of limitations stated by authors were disregarded factors (39%), data quality (28%), unknowable factors (26%), limitations specific to the methods used (22%), data availability (16%), and limited generalizability (8%). 36% of papers did not list any limitations in their discussion or conclusion section. Interpretation Published COVID-19 models were found to be consistently lacking in some of the most important elements required for usability and translation, namely transparency, expressing uncertainty, performance evaluation, stating limitations, and communicating appropriate interpretations. Adopting the EPIFORGE 2020 guidelines would address these shortcomings and improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. We also discovered that most of the operational models that have been used in real-time to inform decision-making have not yet made it into the published literature, which highlights that the current publication system is not suited to the rapid information-sharing needs of outbreaks. Furthermore, data quality was identified to be one of the most important drivers of model performance, and a consistent limitation noted by the modeling community. The US public health infrastructure was not equipped to provide timely, high-quality COVID-19 data, which is required for effective modeling. Thus, a systematic infrastructure for improved data collection and sharing should be a major area of investment to support future pandemic preparedness.

Unified real-time environmental-epidemiological data for multiscale modeling of the COVID-19 pandemic (preprint)

An impressive number of COVID-19 data catalogs exist. None, however, are optimized for data science applications, e.g ., inconsistent naming and data conventions, uneven quality control, and lack of alignment between disease data and potential predictors pose barriers to robust modeling and analysis. To address this gap, we generated a unified dataset that integrates and implements quality checks of the data from numerous leading sources of COVID-19 epidemiological and environmental data. We use a globally consistent hierarchy of administrative units to facilitate analysis within and across countries. The dataset applies this unified hierarchy to align COVID-19 case data with a number of other data types relevant to understanding and predicting COVID-19 risk, including hydrometeorological data, air quality, information on COVID-19 control policies, and key demographic characteristics.

Emergence of an early SARS-CoV-2 epidemic in the United States (preprint)

The emergence of the early COVID-19 epidemic in the United States (U.S.) went largely undetected, due to a lack of adequate testing and mitigation efforts. The city of New Orleans, Louisiana experienced one of the earliest and fastest accelerating outbreaks, coinciding with the annual Mardi Gras festival, which went ahead without precautions. To gain insight into the emergence of SARS-CoV-2 in the U.S. and how large, crowded events may have accelerated early transmission, we sequenced SARS-CoV-2 genomes during the first wave of the COVID-19 epidemic in Louisiana. We show that SARS-CoV-2 in Louisiana initially had limited sequence diversity compared to other U.S. states, and that one successful introduction of SARS-CoV-2 led to almost all of the early SARS-CoV-2 transmission in Louisiana. By analyzing mobility and genomic data, we show that SARS-CoV-2 was already present in New Orleans before Mardi Gras and that the festival dramatically accelerated transmission, eventually leading to secondary localized COVID-19 epidemics throughout the Southern U.S.. Our study provides an understanding of how superspreading during large-scale events played a key role during the early outbreak in the U.S. and can greatly accelerate COVID-19 epidemics on a local and regional scale.

Identifying highly influential travellers for spreading disease on a public transport system (preprint)

The recent outbreak of a novel coronavirus and its rapid spread underlines the importance of understanding human mobility. Enclosed spaces, such as public transport vehicles (e.g. buses and trains), offer a suitable environment for infections to spread widely and quickly. Investigating the movement patterns and the physical encounters of individuals on public transit systems is thus critical to understand the drivers of infectious disease outbreaks. For instance previous work has explored the impact of recurring patterns inherent in human mobility on disease spread, but has not considered other dimensions such as the distance travelled or the number of encounters. Here, we consider multiple mobility dimensions simultaneously to uncover critical information for the design of effective intervention strategies. We use one month of citywide smart card travel data collected in Sydney, Australia to classify bus passengers along three dimensions, namely the degree of exploration, the distance travelled and the number of encounters. Additionally, we simulate disease spread on the transport network and trace the infection paths. We investigate in detail the transmissions between the classified groups while varying the infection probability and the suspension time of pathogens. Our results show that characterizing individuals along multiple dimensions simultaneously uncovers a complex infection interplay between the different groups of passengers, that would remain hidden when considering only a single dimension. We also identify groups that are more influential than others given specific disease characteristics, which can guide containment and vaccination efforts.